Evaluation of alternative design choices for evolutionary mutation testing by means of automated configuration

Mutation testing is a well-established but costly technique to assess and improve the fault detection ability of test suites. This technique consists of introducing subtle changes in the code of a program, which are expected to be detected by the designed test cases. Among the strategies conceived to reduce its cost, evolutionary mutation testing (EMT) has been revealed as a promising approach to select a subset of the whole set of mutants based on a genetic algorithm (GA). However, like any other metaheuristic approach, EMT’s execution depends on a set of parameters (both classical of GAs and context-specific ones), so different configurations can greatly vary its performance. Currently, it is difficult to clarify what are the best values for those parameters by applying manual parameter tuning and whether new design choices could improve its effectiveness with other combinations of values. The experience carried out in this paper applying iterated racing, a well-known automated configuration algorithm, reveals that EMT's performance has been undervalued in previous studies; the new configuration found by iterated racing was able to enhance EMT’s results in all C++ object-oriented programs used in the experiments. This study also confirms alternative design choices as convenient options to improve EMT in this context, namely, detecting and penalizing equivalent mutants by means of Trivial Compiler Equivalence, and learning which mutation operators produced live mutants in the past generations.This work was partially supported by the European Commission (European Regional Development Fund - ERDF), the Spanish Ministry of Science, Innovation and Universities under projects RTI2018-093608-B-C33 and TIN2017-88213-R, the excellence network RED2018-102472-T, the University of Malaga, and Consejería de Economía y Conocimiento de la Junta de Andalucía (grant number UMA18-FEDERJA-003

An experimental and practical study on the equivalent mutant connection: An evolutionary approach

Context: Mutation testing is considered to be a powerful approach to assess and improve the quality of test suites. However, this technique is expensive mainly because some mutants are semantically equivalent to the original program; in general, equivalent mutants require manual revision to differentiate them from useful ones, which is known as the Equivalent Mutant Problem (EMP). Objective: In the past, several authors have proposed different techniques to individually identify certain equivalent mutants, with notable advances in the last years. In our work, by contrast, we address the EMP from a global perspective. Namely, we wonder the extent to which equivalent mutants are connected (i.e., whether they share mutation operators and code areas) as well as the extent to which the knowledge of that connection can benefit the mutant selection process. Such a study could allow going beyond the implicit limit in the traditional individual detection of equivalent mutants. Method: We use an evolutionary algorithm to select the mutants, an approach called Evolutionary Mutation Testing (EMT). We propose a new derived version, Equivalence-Aware EMT (EA-EMT), which penalizes the fitness of known equivalent mutants so that they do not transfer their features to the next generations of mutants. Results: In our experiments applying EMT to well-known C++ programs, we found that (i) equivalent mutants often originate from other equivalent mutants (over 60% on average); (ii) EA-EMT’s approach of penalizing known equivalent mutants provides better results than the original EMT in most of the cases (notably, the more equivalent mutants are detected, the better); and (iii) we can combine EA-EMT with Trivial Compiler Equivalence as a way to automatically identify equivalent mutants in a real situation, reaching a more stable version of EMT. Conclusions: This novel approach opens the way for improvement in other related areas that deal with equivalent versions.This work is partially funded by the European Commission (FEDER), the Spanish Ministry of Science, Innovation and Universities (RTI2018-093608-B-C33), the Spanish Ministry of Innovation and Competitiveness (TIN2017-88213-R), and the University of Malaga (Exhauro project)

Search-Based Mutant Selection for Efficient Test Suite Improvement: Evaluation and Results

Context: Search-based techniques have been applied to almost all areas in software engineering, especially to software testing, seeking to solve hard optimization problems. However, the problem of selecting mutants to improve the test suite at a lower cost has not been explored to the same extent as other problems, such as mutant selection for test suite evaluation or test data generation. Objective: In this paper, we apply search-based mutant selection to enhance the quality of test suites efficiently. Namely, we use the technique known as Evolutionary Mutation Testing (EMT), which allows reducing the number of mutants while preserving the power to refine the test suite. Despite reported benefits of its application, the existing empirical results were derived from a limited number of case studies, a particular set of mutation operators and a vague measure, which currently makes it difficult to determine the real performance of this technique. Method: This paper addresses the shortcomings of previous studies, providing a new methodology to evaluate EMT on the basis of the actual improvement of the test suite achieved by using the evolutionary strategy. We make use of that methodology in new experiments with a carefully selected set of real-world C++ case studies. Results: EMT shows a good performance for most case studies and levels of demand of test suite improvement (around 45% less mutants than random selection in the best case). The results reveal that even a reduced subset of mutants selected with EMT can serve to increase confidence in the test suite, especially in programs with a large set of mutants. Conclusions: These results support the use of search-based techniques to solve the problem of mutant selection for a more efficient test suite refinement. Additionally, we identify some aspects that could foreseeably help enhance EMT

Clang: un compilador de código abierto

En este documento se tratan las versiones 3.4 - 3.6 de Clang.Este documento es un tutorial sobre el compilador Clang para la familia de lenguajes C. En él se introduce Clang y sus características y ventajas, se muestran las opciones más comunes que se le pueden proveer y se presenta un ejemplo detallado de cómo se puede emplear la herramienta para realizar análisis estático sobre el código.16 página

Evolutionary Mutation Testing in Object-Oriented Environments

La prueba de mutaciones es reconocida como un potente método para evaluar la fortaleza de un conjunto de casos de prueba en la detección de posibles fallos en el código. No obstante, la aplicación de esta técnica es costosa, lo cual ha supuesto normalmente un obstáculo para una mayor acogida de la misma por parte de la industria. Varias técnicas han mostrado ser capaces de reducir ampliamente su coste sin mucha pérdida de efectividad, pero también es cierto que estas técnicas solo han sido evaluadas en determinados contextos, especialmente en el ámbito de los operadores de mutación tradicionales para programas procedurales. Por ejemplo, la Prueba de Mutación Evolutiva ha sido aplicada únicamente a composiciones WS-BPEL, a pesar de que se obtuvo un resultado positivo al seleccionar un subconjunto de mutantes a través de un algoritmo evolutivo a fin de mejorar el conjunto de casos de prueba. Como resultado, se desconoce a día de hoy si los mismos beneficios pueden extrapolarse a otros niveles y dominios. En particular, en esta tesis nos preguntamos hasta qué punto la Prueba de Mutación Evolutiva es también útil para reducir el número de mutantes en sistemas orientados a objetos. Más específicamente, nos enfocamos en el lenguaje de programación C++, ya que la prueba de mutaciones casi no se ha desarrollado respecto a este popular lenguaje a juzgar por la falta de artículos de investigación en este campo que se dirigen este lenguaje. Dado que C++ ha sido apenas abordado en cuanto a investigación y en cuanto a la práctica, en esta tesis nos ocupamos de todas las fases de la prueba de mutaciones: desde la definición e implementación de operadores de mutación en un sistema de mutaciones, hasta la evaluación de esos operadores y la aplicación de la Prueba de Mutación Evolutiva entre otras técnicas de reducción del coste. En esta tesis definimos e implementamos un conjunto de operadores de mutación de clase para C++ en MuCPP, herramienta de mutaciones que nos permite llevar a cabo experimentos con programas reales gracias a las características incorporadas a la misma. Estos operadores de mutación son automatizados siguiendo un conjunto de reglas para que produzcan los mutantes que se esperan de los mismos. En términos generales, los operadores de clase generan bastantes menos mutantes que los operadores tradicionales, un porcentaje mayor de mutantes equivalentes y se aplican con diversa frecuencia dependiendo de las características del programa analizado. El desarrollo de reglas de mejora en la implementación de los operadores permite reducir incluso más el número de mutantes, evitando generar mutantes que no son interesantes para el propósito de la prueba de mutaciones. Otro descubrimiento interesante es que el conjunto de mutantes de clase y el de mutantes tradicionales se complementan, ayudando a diseñar un conjunto de casos de prueba más efectivo. También desarrollamos GiGAn, un nuevo sistema para conectar MuCPP y un algoritmo genético para aplicar la Prueba de Mutación Evolutiva a sistemas orientados a objetos en C++. El algoritmo genético permite reducir el número de mutantes que sería generado por MuCPP ya que guía la búsqueda a la selección de aquellos mutantes que pueden inducir a la generación de nuevos casos de prueba (mutantes fuertes). El rendimiento de esta técnica se muestra mejor que el de un algoritmo aleatorio, tanto cuando se buscan diferentes porcentajes de mutantes fuertes como cuando se simula el refinamiento del conjunto de casos de prueba mediante los mutantes seleccionados por ambas técnicas. La estabilidad de la Prueba de Mutación Evolutiva en los diferentes programas analizados y los buenos resultados en aquellos programas de los que se deriva un mayor número de mutantes son observaciones adicionales. Finalmente, realizamos experimentos para evaluar de forma individual a estos operadores de mutación desde una doble perspectiva: cómo de útiles son para la evaluación (TSE) y para la mejora (TSR) de un conjunto de casos de prueba. Para ello clasificamos a los operadores usando dos métricas distintas: el grado de redundancia (TSE) y la calidad para guiar a la generación de casos de prueba de alta calidad (TSR). Siguiendo estas clasificaciones, ponemos en práctica un estudio selectivo teniendo en cuenta que los operadores menos valiosos están en las últimas posiciones. Este enfoque selectivo revela que los operadores no son necesariamente igual de útiles para TSE y TSR, y que estas clasificaciones son apropiadas para llevar a cabo una estrategia selectiva cuando lo comparamos con la aplicación de otras clasificaciones de operadores o la selección aleatoria de mutantes. Sin embargo, favorecer la generación de mutantes individuales a partir de los operadores mejor valorados es mucha mejor opción que descartar operadores al completo debido a que cada uno de estos operadores se centra en una característica concreta del paradigma de orientación a objetos. En conjunto, todas estas evaluaciones en torno a estos operadores de clase sugieren que la naturaleza de los mismos puede limitar los beneficios de aplicar cualquier técnica de reducción del coste.Mutation testing is acknowledged as a powerful method to evaluate the strength of test suites in detecting possible faults in the code. However, its application is expensive, which has traditionally been an obstacle for a broader use in the industry. While it is true that several techniques have shown to greatly reduce the cost without losing much effectiveness, it is also true that those techniques have been evaluated in limited contexts, especially in the scope of traditional operators for procedural programs. To illustrate this fact, Evolutionary Mutation Testing has only been applied to WS-BPEL compositions, despite the positive outcome when selecting a subset of mutants through an evolutionary algorithm with the aim of improving a test suite. As a result, it is unknown whether the same benefits can be extrapolated to other levels and domains. In particular, we wonder in this thesis to what extent Evolutionary Mutation Testing is also useful to reduce the number of mutants generated by class mutation operators in object-oriented systems. More specifically, we focus on the C++ programming language, since the development of mutation testing with regard to this widely-used language is clearly immature judging from the lack of papers in the literature tackling this language. Given that C++ has been hardly addressed in research and practice, we deal with all the phases of mutation testing: from the definition and implementation of mutation operators in a mutation system to the evaluation of those operators and the application of Evolutionary Mutation Testing among other cost reduction techniques. We define a set of class mutation operators for C++ and implement them in MuCPP, which allows us to perform experiments with real programs thanks to the facilities incorporated into this mutation tool. These mutation operators are automated following a set of guidelines so that they produce the expected mutations. In general, class-level operators generate far fewer mutants than traditional operators, a higher equivalence percentage and they are applied with varying frequency depending on the features of the tested program. Developing improvement rules in the implementation of several mutation operators help further reduce the number of mutants, avoiding the creation of uninteresting mutants. Another interesting finding is that the set of class mutants and the set of traditional mutants complement each other to help the tester design more effective test suites. We also develop GiGAn, a new system to connect the mutation tool MuCPP and a genetic algorithm to apply Evolutionary Mutation Testing to C++ object-oriented systems. The genetic algorithm allows reducing the number of mutants that would be generated by MuCPP as it guides to the selection of those mutants that can induce the generation of new test cases (strong mutants). The performance of this technique shows to be better than the application of a random algorithm, both when trying to find different percentages of strong mutants and also when simulating the refinement of the test suite through the mutants selected by each of these techniques. The stability of EMT among different case studies and the good results of the simulation in the programs that lead to the largest set of mutants are additional observations. Finally, we conduct an experiment to assess individually these mutation operators from a double perspective: how useful they are for the evaluation of the test suite (TSE) and its refinement (TSR). To that end, we rank the operators using two different metrics: degree of redundancy (TSE) and quality to guide on the generation of high-quality test cases (TSR). Based on these rankings, we perform a selective study taking into account that the less valuable operators are at the bottom of the classification. This selective approach reveals that an operator is not necessarily as useful for TSE as for TSR, and that these rankings are appropriate for a selective strategy when compared to other rankings or the selection of mutants randomly. However, favouring the generation of individual mutants from the best-valued operators is much better than discarding operators completely because each of the operators targets a particular object-oriented feature. Altogether, these evaluations about class operators suggest that their nature can limit the benefits of any cost reduction technique.Este trabajo fue financiado por la beca de investigación PU-EPIF-FPI-PPI-BC 2012- 037 de la Universidad de Cádiz, por el proyecto DArDOS (TIN2015-65845-C3-3-R) del Programa Estatal de Investigación, Desarrollo e Innovación Orientada a los Retos de la Sociedad del Ministerio de Economía y Competitividad, y por la Red de Excelencia SEBASENET (TIN2015-71841-REDT) del Programa Estatal de Fomento de la Investigación Científica y Técnica de Excelencia del Ministerio de Economía y Competitividad.Número de páginas: 23

Using Evolutionary Mutation Testing to improve the quality of test suites

Mutation testing is a method used to assess and improve the fault detection capability of a test suite by creating faulty versions, called mutants, of the system under test. Evolutionary Mutation Testing (EMT), like selective mutation or mutant sampling, was proposed to reduce the computational cost, which is a major concern when applying mutation testing. This technique implements an evolutionary algorithm to produce a reduced subset of mutants but with a high proportion of mutants that can help the tester derive new test cases (strong mutants). In this paper, we go a step further in estimating the ability of this technique to induce the generation of test cases. Instead of measuring the percentage of strong mutants within the subset of generated mutants, we compute how much the test suite is actually improved thanks to those mutants. In our experiments, we have compared the extent to which EMT and the random selection of mutants help to find missing test cases in C++ object-oriented systems. We can conclude from our results that the percentage of mutants generated with EMT is lower than with the random strategy to obtain a test suite of the same size and that the technique scales better for complex programs

拉古纳大学本科学生学业投入及其对学习成果的影响

The attitude and involvement with which undergraduate university students approach their training process can have a significant impact on the academic results they obtain. The relationship that can occur between low involvement and the possibility of failing or dropping out of studies has led us to propose this research, where the relationship between engagement and academic performance is deepened. The engagement refers to the feeling of well-being of the students with respect to a challenge related to their training process. Students who are studying at university are exposed to living situations that make them more or less committed to their learning process, which can lead to obtain better or worse academic results. This ex-post-facto quantitative study, which was carried out with a sample of 564 first and second year students from the three degrees of the Faculty of Education of the University of La Laguna (Spain), aimed to analyze the relationship between academic commitment and performance in studies. The results reveal that students with higher scores in the different structural dimensions of academic commitment (vigor, dedication and absorption) have higher grades in the subjects they take. The discussion of the study focuses on the need to plan strategies to promote academic commitment in order to strengthen both the trajectories of adaptation and permanence in the degree and the achievement of better academic results in university students.La actitud y la implicación con la que los estudiantes universitarios de grado afrontan su proceso formativo puede incidir de manera significativa en los resultados académicos que obtengan.  La relación que puede darse entre una baja implicación y la posibilidad de fracasar o abandonar los estudios ha llevado a plantear esta investigación, donde se profundiza en la relación entre el engagement y el rendimiento académico. El engagement hace referencia a la sensación de bienestar del alumnado respecto a un desafío relacionado con su proceso formativo. El alumnado que cursa estudios universitarios está expuesto a vivir situaciones que les hacen estar más o menos comprometidos con su proceso de aprendizaje, pudiéndoles llevar a obtener mejores o peores resultados académicos. Este estudio de corte cuantitativo ex-post-facto, que se llevó a cabo con una muestra de 564 estudiantes de primer y segundo curso de los tres grados de la Facultad de Educación de la Universidad de La Laguna (España), tiene como objetivo analizar la relación entre el engagement académico y el rendimiento en los estudios. Los resultados revelan que el alumnado con mayores puntuaciones en las distintas dimensiones estructurales del engagement académico (vigor, dedicación y absorción) presenta mayores calificaciones en las asignaturas que cursa. La discusión del estudio se centra en la necesidad de planificar estrategias promotoras del engagement académico para fortalecer tanto las trayectorias de adaptación y permanencia en la titulación como la obtención de mejores resultados académicos en el alumnado universitario.A atitude e o envolvimento com que os estudantes universitários de licenciatura enfrentam o seu processo de formação pode ter um impacto significativo nos resultados académicos que obtêm.  A relação que pode existir entre o baixo envolvimento e a possibilidade de insucesso ou de abandono justificou esta investigação, na qual se explora a relação entre o engagement e o desempenho académico. O engagement refere-se ao sentimento de bem-estar dos estudantes em relação a um desafio relacionado com o seu processo formativo. Os estudantes que frequentam um curso de ensino superior são expostos a situações que os tornam mais ou menos empenhados no seu processo de aprendizagem, o que os pode levar a obter melhores ou piores resultados académicos. Este estudo quantitativo ex post facto, que foi realizado com uma amostra de 564 estudantes do primeiro e segundo anos dos três cursos de licenciatura da Faculdade de Educação da Universidade de La Laguna (Espanha), tem como objetivo analisar a relação entre o engagement académico e o desempenho no curso. Os resultados revelam que os estudantes com notas mais elevadas nas diferentes dimensões estruturais do engagement académico (vigor, dedicação e absorção) têm notas mais elevadas nas disciplinas que frequentam. A discussão do estudo centra-se na necessidade de planear estratégias para promover o engagement académico, a fim de reforçar não só as trajetórias de adaptação e permanência no curso, como a obtenção de melhores resultados académicos dos estudantes universitários.A atitude e o envolvimento com que os estudantes universitários de licenciatura enfrentam o seu processo de formação pode ter um impacto significativo nos resultados académicos que obtêm.  A relação que pode existir entre o baixo envolvimento e a possibilidade de insucesso ou de abandono justificou esta investigação, na qual se explora a relação entre o engagement e o desempenho académico. O engagement refere-se ao sentimento de bem-estar dos estudantes em relação a um desafio relacionado com o seu processo formativo. Os estudantes que frequentam um curso de ensino superior são expostos a situações que os tornam mais ou menos empenhados no seu processo de aprendizagem, o que os pode levar a obter melhores ou piores resultados académicos. Este estudo quantitativo ex post facto, que foi realizado com uma amostra de 564 estudantes do primeiro e segundo anos dos três cursos de licenciatura da Faculdade de Educação da Universidade de La Laguna (Espanha), tem como objetivo analisar a relação entre o engagement académico e o desempenho no curso. Os resultados revelam que os estudantes com notas mais elevadas nas diferentes dimensões estruturais do engagement académico (vigor, dedicação e absorção) têm notas mais elevadas nas disciplinas que frequentam. A discussão do estudo centra-se na necessidade de planear estratégias para promover o engagement académico, a fim de reforçar não só as trajetórias de adaptação e permanência no curso, como a obtenção de melhores resultados académicos dos estudantes universitários.本科学生在求学过程中的态度及投入会对其学习成果产生较明显的影响。该研究的提出是基于对较低的学业投入与可能导致的学业失败及辍学间关系的考量。通过该研究深入探讨学生学业投入和学习成果间的关系。学业投入是指学生在求学过程中面对相关挑战时所获得的幸福感。大学生所处的环境强迫他们或多或少地致力于自己的学习过程，从而得到或优或差的学业成绩。研究采用事后回溯定量研究，以来自西班牙拉古纳大学教育系三个专业的564名大一大二学生为研究样本，旨在分析学生学业投入及学习成果间的关系。研究结果表明在学业投入结构的不同维度（活力、投入及专注）中获得高分的学生，研修的课程分数更高。该研究讨论的重点集中在探讨建立推动学业投入策略的必要性。有了这样的策略，不仅能加强学生在教育轨迹中的适应性和研读学位的持久性，还能鼓励学生取得更优秀的学业成绩

Automatización de la corrección de prácticas de programación a través del compilador Clang

El proceso de corrección de ejercicios para la enseñanza de lenguajes de programación se ha realizado tradicionalmente de forma manual. En los últimos años se ha avanzado en este aspecto hacia la automatización de la evaluación de las entregas de los alumnos, lo cual viene a mejorar tanto el aprendizaje del alumno como las tareas del profesor. No obstante, uno de los mayores inconvenientes para lograr este objetivo es la limitación en el análisis sintáctico del código para comprobar el cumplimiento de ciertos requisitos en el mismo. Este trabajo muestra cómo el uso de Clang, un compilador maduro de código abierto, puede proporcionar una capacidad de análisis superior mediante el recorrido por el árbol de sintaxis abstracta. Además, se plantea el uso de la prueba de mutaciones como técnica que conciencie a los alumnos de la importancia de diseñar un conjunto completo de casos de prueba para probar sus programas.The process of correction of exercises for the teaching of programming languages has been traditionally accomplished in a manual way. In recent years, several advances have been made regarding this aspect towards the automation of the evaluation of the solutions provided by the students. This support aims to improve both the learning of students and the work of lecturers. However, one of the main drawbacks to achieve this goal is the limitation in the static analysis of the code to check the fulfilment of certain requirements. This paper shows how using Clang, a full-fledged open-source compiler, can provide a greater capacity for analysis through the traversal of the abstract syntax tree. Moreover, the technique known as mutation testing is suggested to make students aware of the importance of designing an adequate test suite to test their applications